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Abstract 

Research  suggests  that  students  who  are  aware  of 
their  own  confusions  and  take  steps  to  resolve 
those  confusions  are  most  likely  to  benefit  from  a 
learning  experience.  At  the  same  time,  there  are 
conversational  maxims,  such  as  Leech’s  politeness 
maxims,  that  may  inhibit  a  student  from  expressing 
and  pursuing  confusions  within  a  tutorial  dialogue. 
We  investigated  students’  expressions  of  confusion 
while  working  through  a  series  of  learning 
activities  with  a  tutor.  We  found  that,  during  the 
times  when  students  were  working  independently 
on  an  activity,  their  expressions  of  confusion  were 
reliable  indicators  of  their  (lack  of)  understanding; 
however,  when  they  were  conversing  with  their 
tutors,  these  same  students  did  not  express 
confusion  and,  in  fact,  the  more  often  the 
expressed  comprehension,  the  worse  they 
performed  on  the  post-test.  This  suggests  that 
student  metacognitive  statements  should  not  be 
interpreted  without  taking  into  consideration  the 
context  in  which  they  were  expressed.  We  briefly 
consider  implications  for  human  tutors  and  the 
development  of  computer  tutoring  systems. 


Introduction 

Twenty  years  ago,  researchers  were  somewhat 
surprised  to  discover  that  students  who 
expressed  confusion  while  studying  worked 
examples  were  more  likely  to  learn  from  that 
activity  than  students  who  appeared  to  easily 


understand  those  examples  (Chi  &  Bassok, 
1989).  The  key  is  in  the  qualifier  “appeared” 
to  understand.  Further  analyses  and  additional 
research  confirmed  that  the  students  who 
expressed  confusion  often  followed  up  by 
taking  steps  to  resolve  that  confusion,  while 
many  of  the  students  who  breezed  through  a 
worked  example  were  simply  not  aware  of  the 
fact  that  they  didn’t  understand  why  each  step 
was  taken  and  how  that  process  might  be 
applied  to  related  but  different  problems  (e.g., 
Ferguson-Hessler  &  de  Jong,  1990;  Pirolli  & 
Recker,  1994;  Renkl,  1997).  In  other  words, 
accurate  metacognition  was  a  necessary 
precursor  to  taking  remedial  action. 

This  work  has  led  to  a  number  of  efforts, 
both  in  the  classroom  and  within  computer- 
based  tutoring  systems,  to  understand  and  train 
metacognitive  skills  such  as  self-explanation 
(e.g.,  Bielaczyc,  Pirolli  &  Brown,  1995;  Chi, 
de  Leeuw,  Chiu  &  LaVancher,  1994;  Conati  & 
Van  Lehn,  2000;  Renkl,  Stark,  Gruber  & 
Mandl,  1998).  Robust  findings  include  the  fact 
that  many  students  do  not  spontaneously  self- 
explain,  but  that  they  can  learn  this  skill  and 
will  benefit  from  this  type  of  training. 

Of  course,  studying  a  worked  example 
independently  is  not  the  same  task  as  working 
through  lesson  materials  with  a  tutor.  In  the 
second  environment,  it  is  possible  that 
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conversational  maxims,  such  as  those  proposed 
by  Leech  (1983),  may  inhibit  certain  types  of 
verbal  expression.  Leech  proposed  that  people 
attempt  to  follow  six  maxims  when  engaged  in 
a  conversation.  The  tact  and  generosity  maxims 
involve  putting  the  interests  and  benefits  of  the 
listener  ahead  of  those  of  the  speaker.  The 
approbation  maxim  involves  minimizing  any 
criticism  and  maximizing  praise  for  the 
listener,  while  the  modesty  maxim  encourages 
the  speaker  to  take  the  opposite  position 
towards  him-  or  herself.  The  agreement  maxim 
suggests  that  people  are  more  likely  to  express 
agreement  than  disagreement  with  their 
listeners.  Finally,  the  sympathy  maxim 
encourages  expressing  sympathy,  rather  than 
antipathy,  towards  the  listener. 

Consider  these  maxims  within  the  context 
of  a  tutorial  dialogue.  Expressing  confusion 
after  a  tutor  attempts  to  explain  a  concept  could 
be  seen  as  a  violation  of  the  approbation 
maxim,  as  this  confusion  could  be  interpreted 
as  a  criticism  of  the  tutor’s  ability  to  explain. 
Thus,  even  if  a  student  is  aware  of  being 
confused  by  something  the  tutor  says,  the 
student  might  not  express  that  confusion  freely. 
If  this  is  true,  then  it  will  have  implications  for 
effective  tutor  strategies  and  it  may  have 
implications  for  the  design  of  computer 
tutoring  systems. 

In  the  rest  of  the  paper  we  present  a  study 
in  which  we  analyzed  transcripts  to  determine 
whether  or  not  students  expressed  confusion 
equally  often  and  with  equal  validity  across 
both  independent  and  communicative  contexts. 

Current  Study 

Data  collection  environment 

A  curriculum  and  learning  environment  were 
created  to  teach  basic  concepts  in  basic 
electricity  and  electronics.  The  four-hour 
curriculum  covered  the  topics  of  complete  and 
incomplete  circuits,  series  and  parallel 
configurations,  voltage  and  fault  detection  in  a 
series  circuit  with  a  multimeter. 

A  screenshot  of  the  learning  environment 
used  to  present  this  curriculum  is  shown  in 


Figure  1.  The  screen  was  divided  into  three 
sections.  The  upper  left-hand  section  contained 
the  primary  lesson  material  (including  didactic 
text,  exercises  and  discussion  questions),  which 
was  presented  in  a  slideshow  fashion.  The 
participants  could  scroll  through  this  section  at 
their  own  pace.  The  upper  right-hand  section 
was  the  circuit  simulator,  which  allowed  the 
participant  to  build  and  manipulate  circuits  to 
test  their  properties.  The  bottom  section  was 
the  message  window  where  the  participants  and 
tutor  interacted  by  typing. 


Figure  1:  Participant  screen. 

The  tutor  and  student  were  not  co-located, 
however  the  tutor  was  able  to  watch  everything 
that  was  happening  in  the  student’s 
environment,  and  gave  feedback  and  whatever 
technical  assistance  and/or  encouragement  that 
he  or  she  deemed  appropriate  through  this 
same  messaging  interface. 

While  working  through  the  curriculum, 
participants  were  instructed  to  direct  all  of  their 
answers,  comments  and/or  questions  to  the 
tutor.  Many  of  the  student  metacognitive 
statements  were  made  within  the  course  of  a 
multi-turn  dialogue  (often  centered  on  a 
discussion  question  embedded  in  the  lesson). 
Flowever,  there  were  times  when  the  student 
was  reading  a  slide  or  building,  observing  and 
measuring  circuits  in  the  workspace  and 
spontaneously  sent  a  metacognitive  statement 
to  the  tutor,  such  as,  “Oh!  I  see!”  We  refer  to 
the  first  type  as  a  metacognitive  statement 


made  within  the  context  of  a  dialogue  with  the 
tutor,  and  the  second  type  as  a  metacognitive 
statement  made  within  the  context  of  an 
independent  activity. 

Procedure 

After  completing  informed  consent  paperwork, 
participants  filled  out  a  demographic 
questionnaire  and  took  a  38  item  multiple 
choice  pre-test.  They  were  introduced  to  the 
tutor  and  given  a  short  demonstration  of  how  to 
work  in  the  learning  environment.  The 
majority  of  the  experimental  session  was  spent 
with  the  student  working  through  the  lesson 
material  in  the  learning  environment.  At  the 
end  of  the  experiment,  participants  completed  a 
2 1  item  multiple  choice  post-test  and  a  reaction 
questionnaire.  They  were  then  debriefed  and 
excused. 

Corpus 

The  corpus  includes  dialogues  from  each  of 
thirty  participants  distributed  across  three 
experienced  tutors.  The  average  age  of  the 
participants  was  22.4  years  (standard  deviation 
=  5.0)  and  exactly  half  of  them  were  male.  The 
entire  corpus  includes  8,085  dialogue  turns 
taken  by  the  student  and  tutor,  and  56,133 
tokens  (words  and  punctuation). 

Coding 

Two  independent  raters  identified  and  coded 
the  metacognitive  statements  made  by  the 
student  and  the  tutor  with  very  high  reliability 
(kappa  =  0.99).  Metacognitive  statements  were 
defined  as  statements  that  contained  the 
speaker’s  feeling  about  his  or  her 
understanding,  but  did  not  include  domain 
content.  Metacognitive  statements  were  further 
classified  as  positive  or  negative.  Positive 
metacognitive  statements  were  defined  as 
statements  that  expressed  understanding  (i.e.  “I 
got  it”,  “That  makes  sense”,  etc),  whereas 
negative  metacognitive  statements  expressed 
confusion  (i.e.  “1  don’t  understand”,  “1  am 
confused”,  etc). 


Results 

Overall,  students  made  significantly  more 
positive  metacognitive  statements  (mean  = 
12.9,  standard  deviation  =  8.3)  than  negative 
metacognitive  statements  (mean  =1.8,  standard 
deviation  =  2.0),  paired  t( 29)  =  8.7 ,p  <  0.05. 

The  total  number  of  metacognitive 
statements  made  across  the  entire  dialogue  was 
significantly  negatively  correlated  to 
performance  on  the  post-test  (r  =  -0.48,  p  < 
0.01).  In  addition,  both  the  number  of  positive 
metacognitive  statements  and  the  number  of 
negative  metacognitive  statements  were 
significantly  negatively  correlated  to 
performance  on  the  post-test  (r  =  -0.46,  p  < 
0.05  and  r  =  -0.29, p  <  0.05,  respectively). 

As  explained  earlier,  the  metacognitive 
statements  were  divided  into  those  made  within 
the  context  of  a  dialogue  with  the  tutor  and 
those  made  spontaneously  by  the  student  as  he 
or  she  conducted  an  independent  activity. 

Most  of  the  metacognitive  statements  were 
made  within  during  ongoing  content-based 
dialogues  with  the  tutor.  Within  this  context, 
students  made  positive  metacognitive 
statements  (average  =  10.9,  standard  deviation 
=  6.7)  significantly  more  often  than  negative 
ones  (average  =  0.5,  standard  deviation  =  0.78), 
paired  t( 29)  =  8.8,  p<  0.01.  In  addition,  in  this 
context,  the  number  of  positive  metacognitive 
statements  made  per  student  was  significantly 
negatively  correlated  with  post  test 
performance,  r  =  -0.51  ,P  <  0.01,  however  the 
number  of  negative  metacognitive  statements 
made  was  not  correlated. 

Considering  the  metacognitive  statements 
that  the  students  made  within  the  context  of 
conducting  an  independent  activity,  there  was 
no  statistically  significant  difference  in  the 
average  number  of  positive  and  negative 
metacognitive  statements  made  per  student 
(average  =  1.4,  standard  deviation  =  1.8  and 
average  =  1.3,  standard  deviation  =  1.6, 
respectively). 

In  addition,  in  this  context  the  number  of 
negative  metacognitive  statements  made  per 
student  was  significantly  negatively  correlated 
with  post  test  performance,  r  =  -0.41, <  0.05, 


however  the  number  of  positive  metacognitive 
statements  made  was  not. 

Finally,  the  number  of  negative 
metacognitive  statements  spontaneously  made 
by  students  during  independent  activities  was 
not  correlated  with  the  number  of 
metacognitive  statements  made  within  the 
context  of  the  tutorial  dialogue,  however  it  was 
correlated  with  the  number  of  positive 
metacognitive  statements  made  during 
dialogue,  r=  0.63 ,p  <  0.01. 

Discussion 

Our  results  showed  that,  when  working 
independently  (reading  lesson  slides  or 
completing  an  activity  within  the  circuit 
simulation  workspace),  students  were  equally 
likely  to  spontaneously  generate  statements 
indicating  confusion  as  they  were  to  generate 
statements  indicating  comprehension.  In 
addition,  the  statements  indicating  confusion 
appeared  to  be  meaningful,  as  the  more  of 
these  statements  a  student  made,  the  lower  that 
student  scored  on  the  post-test.  This  may  seem 
to  conflict  with  previous  research,  but  a  more 
likely  (and  unfortunate)  explanation  is  that 
accurate  metacognition  is  a  necessary  but  not 
sufficient  condition  to  yield  effective  learning, 
and  our  students  may  not  have  been  able  to 
resolve  their  confusions  as  well  as  we  would 
have  hoped. 

In  contrast,  these  same  students  rarely 
expressed  confusion  within  the  context  of  a 
dialogue  with  their  tutor.  In  fact,  the  more 
often  a  student  expressed  confusion  during  an 
independent  activity,  the  more  often  that 
student  expressed  comprehension  during  a 
dialogue. 

It  should  be  noted  that  this  cannot  be 
completely  explained  by  pointing  to  the 
remedial  dialogues  that  the  tutor  initiated  after 
the  students’  statements  of  confusion,  because 
of  the  large  disparity  in  raw  frequency  counts. 
Recall  that  students  only  made  an  average  of 
1.3  negative  metacognitive  statements  during 
independent  activities,  but  made  an  average  of 
10.9  positive  metacognitive  statements  during 
their  dialogues  with  tutors. 


These  findings  suggest  that  there  is 
something  fundamentally  different  between  the 
context  of  working  independently  and 
interacting  with  a  tutor.  We  propose  that 
linguistic  conventions,  such  as  Leech’s 
principles  of  politeness,  may  constrain  a 
student’s  willingness  to  express  confusion  to  a 
tutor. 

It  appears  as  if  tutors  should  be  suspicious 
of  any  student  statements  expressing 
comprehension  made  within  a  dialogue  context 
and  should  probably  rely  on  other  means  of 
assessing  student  comprehension.  On  the  other 
hand,  tutors  may  be  able  to  take  student 
statements  of  confusion  made  during 
independent  activities  seriously  and  initiative 
effective  remedial  strategies. 

It  is  somewhat  of  an  open  question  how 
faithfully  results  found  with  human  tutors  are 
generalizable  to  computer-based  tutoring 
systems.  An  important  next  step,  therefore,  is 
to  attempt  to  replicate  these  results  with  a 
computer-based  tutoring  system.  However, 
should  these  results  generalize,  system 
developers  will  need  to  implement  tutorial 
strategies  that  accommodate  the  unreliability  of 
student  metacognitive  reports  given  within  the 
context  of  the  tutorial  dialogue.  For  example, 
instead  of  asking  if  the  student  understands, 
tutors  could  ask  content-based  questions  to 
gauge  a  student’s  comprehension.  Of  course, 
the  data  presented  here  do  not  speak  to  the 
effectiveness  of  this  strategy,  and  research  is 
also  needed  to  determine  the  most  effective 
pedagogical  strategy. 

While  we  have  presented  some  data 
suggesting  that  conversational  maxims  are  in 
competition  with  certain  effective 
metacognitive  learning  skills  within  a  tutorial 
dialogue  context,  there  were  several  limitations 
to  this  study  that  must  be  taken  into  account. 
One  issue  is  that  only  typed  comments  sent  to 
the  tutor  were  evaluated,  and  other  indicators 
of  confusion  (for  example,  facial  expressions) 
were  not  captured.  In  addition,  the  coding 
system  applied  to  the  transcripts  only  allowed  a 
student  turn  to  be  coded  as  a  pure 
metacognitive  statement  or  a  statement  dealing 


with  domain  content.  It  is  likely  that  some 
statements  can  serve  both  roles  simultaneously. 
Our  analyses  were  limited  to  those  statements 
that  were  “pure”  metacognitive  and  did  not 
include  any  domain  content. 

In  addition  to  overcoming  these  limitations, 
future  research  is  necessary  to  determine  the 
generalizability  of  this  finding  and  appropriate 
strategies  for  overcoming  this  situation. 
Finally,  this  work  only  touched  on  a  small 
portion  of  Leech’s  conversational  maxims,  and 
further  explorations  into  how  they  may  support 
or  undermine  the  goals  of  a  tutorial  dialogue 
could  be  very  instructive. 
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